Field of the Invention
The present invention relates to an image processing apparatus to which a search technique for searching corresponding points or a motion vector between moving image frames is applied, a control method therefor, and a storage medium storing a control program therefor.
Description of the Related Art
Image processing apparatuses employed in image pickup apparatuses in recent years require advancement of vector search performance (a search possibility rate, annon-outlier rate, and accuracy) because the number of pixels increases and performance is improved. The search possibility rate indicates a ratio of possibility that only one peak is obtained on a correlation value map. The non-outlier rate is a ratio in which a motion vector that is output by template matching (TM) as a correct result is not incorrect greatly. The accuracy is a parameter about slight deviation of the motion vector from a true motion vector that is not as large as the outlier.
On the other hand, the TM holds an important position in searching corresponding points or a motion vector between moving image frames and in tracking a locus of a feature point (hereinafter referred to as a “motion vector”). Motion vector information obtained by the TM is utilized as input information required for applications of an image pickup apparatus like multi-image compositions including image stabilization, a multi-image noise reduction (NR), and a dynamic range expansion (HDR).
In a design of a motion vector search unit by the TM, a search range, a template size, and a determining threshold are main fundamental-planning values that influence performance.
The search range is determined on the basis of a moving amount of an image to be searched between frames.
The template size is determined so as to keep the search possibility rate high and to keep the outlier rate low. In order to simultaneously keep performance and robustness, for example, in order to correspond to a large camera shake, it is necessary to search a wide search range. In order to determine the only one peak within the wide search range, a large template size is required so that a wide area may be caught without being influenced by a repeated pattern, a flat part, etc.
However, when the template size is enlarged in order to correspond to a wide search range, accuracy tends to deteriorate. The reason why is because when a partial area other than a remarkable point deforms and moves with respect to the remarkable point in the template and the partial area includes a remarkable feature, the accuracy of the motion vector search deteriorates due to influence of the relative movement. Specifically, the accuracy deteriorates due to competition of parallax owing to an arrangement of objects, deformation caused in a moving body, distortion of an optical system, or an effect of rolling shutter distortion of an image pickup device. Naturally, the accuracy tends to deteriorate due to the above-mentioned deformation or distortion as the template size becomes larger.
A determining threshold that is used to determine a search possibility and an outlier is difficult to be defined arbitrarily for the purpose of improved efficiency, because the threshold is determined depending on an object image in consideration of a proper elimination rate, and an acceptance rate of an outlier.
In recent years, there is a proposed method that improves search performance (a search possibility rate, a non-outlier rate) by reducing a search range in addition to reduction of a search computation amount. Moreover, since the reduction of a search range tends to decrease a template size, improvement in accuracy is also expectable. For example, in a case where a motion vector caused by a motion of an object image due to a motion of an image pickup apparatus is searched, the motion of the object image in frames can be reduced by using a camera shake correcting function of the image pickup apparatus. The reduction of an assumed motion of an image between search frames enables to narrow the search range (see Japanese Laid-Open Patent Publication (Kokai) No. H8-275049 (JP H8-275049A)).
Moreover, rough positions of corresponding points are specified on the basis of information combining position-posture information detected with a gyro sensor, focal length information, etc., and fragmentary information about a motion of an image caused by a camera shake of the image pickup apparatus. Furthermore, there is a proposed technique that moves a center position of a search range to narrow a search range (see Japanese Laid-Open Patent Publication (Kokai) No. 2009-49457 (JP 2009-49457A)).
When a search range is simply determined under a condition where an effect of an optical image stabilization is maximized, and when a camera shake momentarily exceeds a movable range of an optical image stabilizing mechanism and a margin range of an electronic image stabilization, a motion vector protrudes from the search range. On the other hand, when a search range is determined according to the worst case, the effect of the search range restriction is not obtained.
Moreover, even if the existing ideas described in the background art and the techniques of the above-mentioned two publications are merely combined, the improved efficiency expected is not obtained. For example, a motion of an image is reduced with the image stabilizing function of the image pickup apparatus shown in JP H8-275049A. In addition, when the search position control on the basis of the position-posture information that does not assume to use the optical image stabilization shown in JP 2009-49457A is combined, the reduction of the motion of the image with the optical image stabilization conflicts with the rough prediction positions of the corresponding points indicated by the motion vector that is obtained from the position-posture information. As a result, the effect of the search range restriction is not obtained. This becomes remarkable as the optical image stabilization effect is raised by increasing the damping rate of the optical image stabilization.